AD659773 


MEMORANDUM 

RM-5346-PR 

SEPTEMBER  1987 


RELIABILITY  ASSESSMENT  IN 
THE  PRESENCE  OF  RELIABILITY  GROWTH 

A.  J.  Gross  and  M.  Kamins 


n  D  C 

n'rJrn 


.1 _ 


m  OCT  1  9  1967 

i  '  i  j 

Li  ui  i -ID- 

PREPARED  FOR:  q 

UNITED  STATES  AIR  FORCE  PROJECT  RAND 


7 ^ 


R-flnD 


(?,axfea'iatioK 


5ANTA  MONICA  •  CALIFORNIA- 


Best  Available  Copy 


(• 


MEMORANDUM 

RM-5340-PR 

SEPTEMBER  1967 


RELIABILITY  ASSESSMENT  IN 
THE  PRESENCE  OF  RELIABILITY  GROWTH 

A.  J.  Gross  and  M.  Kamins 


Thi-  research  is  supported  !ty  the  United  States  Air  Forre  under  Project  HAND— Con¬ 
tract  No.  h  1  lf>20-67-C-00  15— monitored  hv  the  Directorate  of  Operational  Requirements 
and  Development  Plans.  Deputy  Chief  of  Staff.  Research  and  Development.  Hq  L'SAF. 
\  lews  or  conclusions  contained  in  this  Memorandum  should  not  be  interpreted  as 
representing  the  official  opinion  or  policy  of  the  United  States  Air  Force. 

DISTRIBUTION  STATEMENT 

Distribution  of  this  document  is  unlimited. 


7 & 


-  i  i  i  - 


PRKFACK 

Tliis  Memorandum  is  another  expression  of  RAND's  long  term  interest 
in  and  involvement  with  reliability  assessment:  of  Air  Foi ee  weapon 
systems,  past,  present,  and  future,  It  provides  a  methodology  for 
estimating  past,  current,  and  neai-term  future  reliability  lor  systems 
that  can  be  shown  to  improve  in  launch  and/or  in- flight  reliability 
during  Llieir  development  and  early  operational  phases.  This  method¬ 
ology  should  be  directly  useful  to  persons  responsible  for  specifying, 
from  actual  test  results,  the  appropriate  reliability  values  to  be  used 
lit  target  ing  and  requirements  studies.  It,  may  also  be  helpful  to  those 
involved  in  cost-effectiveness  evaluations  during  development  and  early 
operational  periods,  in  addition  to  being  of  interest  to  mathematicians, 
statisticians,  operations  researchers,  and  some  pi  eject  or  development 
engi neers , 
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SUMMARY 


The  relatively  brief  bistory  of  rocket  vehicle,  and  particularly 
ICliM  development,  lias  caused  a  rediscovery  of  one  of  the  better-known 
features  of  the  fly- fix- fly  methods  of  aircraft  devf:lopinent--tlie 
systems  rend  to  become  more  reliable  as  one  gains  experience  and 
applies  it  to  design  improvement.  Since  changes  in  reliability  have 
important  implications  for  those  involved  in  planning,  procurement, 
support,  and  command,  a  meLhod  for  assessing  this  changing  reliability 
at  any  given  stage  or  projecting  it  to  near- luture  time  periods  should 
be  of  considerable  use. 

Tills  Memorandum  proposes  four  reliability  growth  models  oi 
patterns  that  can  be  titled  to  acLual  experience  data  (i.c.,  launch 
or  flight-test  results)  to  discern  the  quantitative  characteristics 
of  the  growth  within  relatively  well-defined  tolerances.  This  ob¬ 
jective  is  achieved  by  defining  appropriate  parametric  models  and 
subsequently  using  maximum  likelihood  procedures  to  obtain  estimates 
ul  the  parameters,  and  hence  of  the  reliability.  The  models  are 
studied  in  detail  with  regard  to  their  ability  to  meet  sutlieicnt 
conditions  for  the  existence  oi  maximum  likelihood  estimators,  and 
it  is  shown  that  only  two  oi  them  yield  maximum  likelihood  estimates 
that  can  be  used  multi  the  most  general  circumstances.  'Numerical 
procedures  are  developed  lor  obtaining  the  estimates  oi  the  parameters. 
Further,  the  vai i aikt- covai land:  matrix  of  the  estimates  Is  used  Lo 
construct  approximate  coni idence  regions. 
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These  models  arc  compared  with  each  other  and  with  alternative 
nonparametr 1c  and  Bayesian  approaches,  using  simulated  data  to  make 
the  comparisons.  These  comparisons  show  that  under  the  conditions 
scl  forth  in  this  study,  three  of  the  parametric  models  are  generally 
superior  in  their  predictive  and  assessment  characteristics  to  repre¬ 
sentative  nonparauie t r i c  methods ,  and  to  an  applicable  Bayesian  pro¬ 
cedure.  However,  none  of  t.hese  three  parametric  models  is  universally 
applicable,  since  the  desirable  qualify  of  minimum  bias  can  be  achieved 
only  by  deciding  beforehand  whether  the  system  reliability  is  tending 
closely  enough  to  the  usually  unattainable  goal  of  1.0,  or  perfection, 
and  choosing  the  model  appropriate  to  the  circumstances. 
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I.  INTRODUCTION 


The  reliability  of  a  new  weapon  system  is  a  critical  determinant 
of  its  effectiveness,  and  is  thus  of  vital  interest  to  those  in  the 
Air  Force  concerned  with  strategic  planning,  procurement,  system  sup¬ 
port,  and  operational  command.  The  introduction  of  long-range  ballistic 
missiles  into  the  Air  Force  weapons  arsenal  has  accelerated  the  obso¬ 
lescence  of  the  informal  treatment  of  weapon  system  reliability 
characteristics  which  proved  reasonably  satisfactory  for  manned 
aircraft.  The  shortcomings  were  recognized  before  the  missile  age, 
and  probably  arose  from  experiences  with  electronic  equipment  in  the 
years  following  World  War  II.  By  the  time  the  first  generation  of 
ballistic  missiles  was  entering  military  use,  the  notion  of  a  mean 
time  between  failures  (MTBF)  had  seen  wide  usage  in  assessments  of 
alert  capability  and  support  requirements,  particularly  for  radars 
and  systems  requiring  continuous  operation.  The  ’’one-shot”  aspect 
oi  ballistic  flight  lias  renewed  interest  in  Lhe  reliability  of  inde¬ 
pendent.  binomial  or  Bernoulli  trials,  but.  what  seems  to  be  a  new 
feature  of  these  systems  has  added  another  dimension  to  the  problem. 

In  May  cf  1964,  Space  Technology  Laboratories  (STL,  now  known  as 
Systems  Division  oi  Thompson-Ramo-Wooldridge)  published  the  results 
of  a  study  done  under  contract  to  NASA,  Reliability  Growth  oi  U.S. 
Rockets  (U)  [9].  The  study  analyzed  the  flight  test  results  from 
nine  separate  rocket  vehicle  programs  (including  four  Air  Force 
ballistic  missiles),  and  showed  that  each  one  enjoyed  a  substantial 


reliability  growth  during  its  development  and  early  operational 
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stages,  In  an  unc las sified  portion  of  their  report,  they  concluded 
that 


The  proportion  of  successful  flights  in  a  program, 
is  an  indication  of  the  vehicle's  average  reliability 
for  the  program.  However,  the  average  reliability  of 
past  flights  is  not  satisfactory  for  estimating  present 
reliability  or  for  predicting  the  reliability  of  future 
flights  of  vehicles. 

If  the  reliability  is  increasing  from  flight  to 
flight,  the  average  reliability  will  lie  somewhere  be¬ 
tween  the  true  reliability  of  the  first  flight  and  the 
true  reliability  of  the  last  flight,  where  both  of  the 
true  reliabilities  arc  unknown.  I  resent  and  future 
reliabilities  could  be  grossly  underestimated  by  assuming 
the  average  reliability  of  the  past.* 


STL  developed  such  a  prediction  model,  and  applied  it  to  data 
from  the  flight  tests  of  (among  others)  the  Air  Force's  Minuteman 
ICBM,  using  only  the  early  launches  of  this  system.  The  projection 
of  future  reliability  made  from  that  analysis  correctly  predicted, 
w.thin  narrow  limits,  the  outcome  of  Minuteman  flight  tests  performed 
throughout  the  following  two  years.  Unfortunately,  we  cannot  evaluate, 
with  regard  to  subsequent  performances,  most  of  the  other  systems 
whose  launch  results  t»TL  analysed.  Many  of  twom  n^u  been  removeu 
from  operational  service  by  the  time  the  report  was  published,  and 
others  followed  soon  after.  The  data  for  those  that  remained  opera¬ 
tional  are  unavailable  for  this  study.  Two  other  Air  Force  systems 
for  which  we  have  subsequent  flight  test  data  show  mixed  experience. 

For  one  the  STL  prediction  was  excellent,  but  an  equally  good  one 
could  have  been  generated  by  merely  omitLing  the  I<  &  I)  and  early 


>v 

Reference  9, 


pages  4-1,  4-2  (U). 
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opciational  launches;  for  the  oLher,  subsequent  operational  results 
fell  too  far  from  the  original  prediction  Lo  bo  considered  a  valida¬ 
tion,  but  the  program  was  sufficiently  unusual  in  other  respects  to 
make  it  unsuitable  for  tests  of  predictive  models  anyway. 

The  purpose  of  this  study  is  to  explore  the  utilization  of 
various  pammetric  reliability  growth  models  in  evaluating  current 
reliabilities  and  predicting  near-term  future  reliabilities  of  com¬ 
plex  weapon  systems  that  exhibit  reliability  growth  during  their 
deve  1  opulent . 

The  primary  emphasis  in  this  study  has  been  on  parametric  re¬ 
liability  growth  models  rather  than  on  nonparametr ic  ones  for  the 
following  reasons: 


1.  Lower  confidence  bounds  associated  with  nouparametric 
reliability  growth  models  arc  very  conservative.  Since 
important  decisions  are  made  on  the  basis  of  a  lower 
confidence  bound,  unnecessary  penalties  imposed  upon 
this  quantity  cause  indefensibly  higher  costs  with 
virtually  no  added  capability. 

2.  Although  criticism  of  parametric  methods  is  generally 
hard  to  refute  on  a  theoretical  basis,  parametric 
techniques  are  oiten  quite  useful  in  piucirce.  Use 
o£  a  nonparanieLric  approach  should  be  contingent  upon 
finding  a  satisfactory  substitute  for  parametric 
metnods  in  hand. 

3.  There  is  analytical  suppoi-t  for  the  expectation  of  an 
exponential  reliability  growth  characteristic  in  a 
process  which  is  a  reasonable  facsimile  of  weapon 
development  testing  [8].  In  addition,  there  is  con¬ 
siderable  empirical  material  indicating  that  for  such 
processes,  reliability  is  generally  increasing,  ttuu 

at  a  progressively  decreasing  rate  [7],  [8],  [9],  [10],  [11]. 

We  have  not  completely  ignored  other  methods  of  calculating 
lower  confidence  bounds  for  reliability  growtli  models.  Lor  instance, 
the  Appendix  develops  a  Bayesian  procedure  for  obtaining  confidence 
intervals  for  the  reliability  growth  at  eacli  stage.  The  Bayesian 
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procedurc  takes  into  account  the  "exogenous"  information  with  regard 
to  the  system.  Wc  later  compare  the  conCider.ee  intervals  obtained  by 
both  the  Bayesian  approach  and  another  nonparumetric  method  to  those 
obtained  by  the  parametric  approach.. 

The  reliability  growth  models  cons  Leered  for  this  study  assume 
that  the  weapon  system's  reliability  during  the  k-th  stage  of  testing 
is  a  function  of  the  ultimate  re  j.  iabi  Li.  ty  that  would  be  attained  if 
the  number  of  stages  is  s.ifowcd  to  approach  infinity,  and  one  or  more 
parameters  modifying  the  rate  of  rel  lubi  i.j.ty  growth.  Specifically, 
v2  consider  first  of  all  reliability  g.owth  models  of  t.hc  form 

(1)  \  =  Rw  -  «F(k), 

wi.ere  is  the  weapon  system's  reliability  at  the  k-th  stage  of 
development,  R  is  the  ultimate  system  reliability,  #>  0  Is  a  pa¬ 
rameter  L'naL  quantifies  the  amount  of  growth  occurring  between  stages 
1  and  aq  and  i'(k)  is  a  positive  decreasing  function  of  ,  character¬ 
izing  that  growth.  Lloyd  and  Lipow  p6]  discuss  one  tm  mbei  of  this 
class  of  models  and  pay  specific  attention  to  the  cate  for  which 
F(k)  =  l/k,  The  second  class  of  growth  models  considered  is  the 
exponential  class,  namely, 

-ur..k 

00  \  •-  1  -  t/,  e  z  , 

wliere  lias  the  same  meaning  as  in  (1),  oe  is  a  parameter  indicating 
the  amount  of  reliability  growth,  0  <  rI}  &  ,  and  cr  >  0  is  a  parameter 

measuring  tl.  into  of  reliability  growth.  Xlu.6  inode!  is  developed  and 
ut.ilized  In  [9]. 


:,:l  |,l,'l,l,;|l:  Lillll'!!::::!!liiiii.iiwi *>■< ■  irilfalliiaJ.  lliil-!!' 
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In  a  sense,  a  third  class  of  parametric  models  can  be  considered 
a  general ’’.nation  of  (1)  and  (2)  and  is  represented  as 

(3)  Id  •=  R  -  Q'k, 

K  CD 


where  and  have  the:  same  meanings  as  in  (1),  and  o'  is  a  real 
number  that  lies  in  the  open  interval  (0,  1).  a  indicates  the  total 
amount  ol  growth,  i.e.,  the  growth  from  stage  1  to  stage  co. 

The  fourth  and  final  class  of  parametric  models  treated  here  is 
given  by 

-Gv/k 

(4)  Rj,  “  o-jO  , 


where  0  &  Si  and  os,  >  0. 

Section  II  of  this  Memorandum  develops  the  estimation  procedures 
foi  reliability  growth  models,  describes  the  pertinent  restrictions, 
and  develops  the  equations  that  permit  the  estimation  of  currcnL  and 
rn.ar-term  reliability,  as  well  as  the  confidence  bounds  on  the  esti¬ 
mates.  Section  111  displays  the  behavior  of  the  two  models  that 
appear  most  promising  for  reliability  assessment  in  the  presence  of 
reliability  growth,  stresses  tbc  shortcomings  oi  each,  and  shows 


Models  3  and  A  suffer  from  a  number  of  important  shortcomings, 
among  which  are  (1)  computational  (convergence)  dii iiculties ,  (2)  a 
higher  variability  ol  prediction,  and  (3)  the  necessity  to  check  lor 
the  conditions  which  guarantee  a  unique  maximum  (at  times,  these  con¬ 
ditions  at c  not  met).  Although  analysis  programs  were  written  and 
exercised  for  both  these  models,  the  results  were  Sufficiently  inferior 
to  those  (or  Models  1  and  2  (and  Lhe  variations  thereon)  LhaL  no 
detailed  examination  of  those  resulLs  lias  been  given  hero.  However, 
the  Appendix  does  present  Lhc  analytical  developments  lor  all  four 
in. -dels  - 


0 


some  comparisons  between  the  parametric  and  otlier  methods.  Section  IV 
gives  the  conclusions  oi  the  study. 

The  mathematically  sophisticated  reader  should  have  no  difficulty 
in  understanding  any  portion  of  the  discussion  that  follows.  Those 
whose  background  does  not  include  Lraining  in  probability  theory  and 
statistics  may  find  certain  equations  in  Sec.  II  quite  difficult  to 
understand,  but  might  be  well  advised  to  follow  the  narrative  in  any 
case.  Should  even  this  prove  too  tedious,  the  reader  may  prefer  to 
turn  directly  to  the  somewhat  less  mathematically  taxing  discussion 
of  the  models'  experimental  behavior  in  Sec.  HI,  and  accept  the 
allegations  made  there  concerning  Lhc  developments  in  Sec.  II. 
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IT.  T1IK  MAXIMUM  TJKKI.THOOI)  METHOD  I'OR  ESTIMATING 
PARAMETERS  OF  KKMABTI.ITY  GROWTH  MODELS 

GENERAL  DISCUSSION 

The  general  analysis  of  .’reliability  growth  models  for  complex 
weapon  systems  proceeds  in  the  following  manner:  A  test  program  is 
conducted  in  N  stages;  at  Lhe  k-Lh  stage,  successes  are  recorded 
in  n.  trials.  When  the  final  or  N-th  stage  is  completed,  we  want  to 
lit  to  the  data  a  growth  curve  which  then  i.3  used  to  evaluate  current 
reliabilities  and  predict  near-term  future  reliabilities. 

The  general  parametric  reliability  growth  function  can  be  written 
as 

(5)  \  “  {(Rw,  ar  or2,  ....  cvp;  k) , 

where  R^  and  liave  the  same  meanings  as  in  (11,  and  ctj  ,  .  .  .  ,  a  arc 

p-parameters  determining  the  growth  ct  reliability  from  stage  t.o  stage. 

The  vector  (R_,  aki  • ■ • >  cr  )  is  constrained  to  lie  in  a  convex  region 

r.  As  a  first  step  in  estimating  reliability  growth,  estimates  are 

required  of  the  (p  +  1)  parameters  R  ,  <*,  ,  a  .  This  is  carried 

00  i  1' 

out  in  general  by  the  method  of  maximum  likelihood,  whose  estimators 
are  used  primarily  because  of  their  favorable  large  sample  properties. 

THE  METHOD  OR  MAXIMUM  I, IKELTHOOD 
At  the  k-th  stage  of  testing 


-8- 


where  pr|x  c  s^|  is  the  probability  of  exactly  successes  in 
trials.  Using  (5)  and  assuming  that  the  test  stages  are  statistically 
Independent,  the  likelihood  function  whose  logarithm  is  to  be  maximized 
is  given  by 


x  -  n  l 

k-l  Vs 


ffR,,  alf  ....  orp;  k)]  k[l  -  f(Ra>1  ai . op;  k)pk  Sk. 


then  require  the  set  of  values  (R  ,  a.  ,  .  .  .  ,  a  )  which  lies 

CD  1  P 

in  the  region  so  that  0  £  f(R  ,  a,  ,  ■  •  •  ,  a  i  k)  :£  1  and  which  maximizes 

co  1  p 


logc  X  in  ttiis  region.  To  set  a  unique  maximum  by  the  standard  tech¬ 
nique  of  partial  differentiation  oi  log^  X  with  respect  to  R^,  o.  , 
a Cfp»  it  is  sufficient  to  demonstrate  that  log^  X  is  a  strictly 
concave  function  of  these  parameters,,  and  that  the  maximum  occurs  in 
the  interior  of  F,  i.e.,  the  maximum  occurs  in  the  region  0  < 
f(R„,  c^,  •••,  ap;  k)  <  1,  tor  each  k. 

A  sufficient  condition  that  loge  X  be  a  strictly  concave  function 
is  that  the  matrix 


a2  i°gc  x 

TTi 


a2  ioge  x 


aR*a"P 


32  logc 

X 

bZ  loge 

aR.a“l 

aR«a°p 

b2  logc 

X 

3 2  1«SC 

ber^ 

3a,  dan 
*  P 

b2  logc 

X 

_  2  , 

b 

dot-,  da 

1  P 

“  .  2  " 

.  V 

'  "  n 
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be  negative  definite.  Thus,  assuming  that  log^  Z  is  concave, 
f (R  ,  a.  ,  ....  a  ;  k)  is  a  differentiable  function  of  (R  ,  ct,  ,  •  ■  •  ,  a  )  , 

CD  1  p  CO  X  P 

and  that  the  maximum  occurs  in  the  ulterior  of  T,  then 


(7)  loge  X  =  const.  +  2^  sk  loge  f  (K^,  ,  ....  a k) 


4  iC  (nk  "  Sk)  loge  ^  ‘  *  (Ra=’  al’  ap:  k^  * 


3  log,  x  y*  Vr^V  al’  •••’  V  k> 

SR^  k=l  l(ROT>  O']*  •••>  ap5 


X  (nk  •  "k)fK  <R-’  <V  •••>  V  k) 

-  y  - ~ - : _ 

k^i  t1  -  5-(Rro>  «!■  ••••  «  ;  k)] 


_  ,  „  N  s.  f  ( R  ,  a,  ,  •  ■  .  ,  a  ;  k) 

3  logc  X  ^  k  «°  1  p’  ' 

3aj  fe  f(R».  »i  -  •  •  •  >  v  M 


N  (nk  -  Sk)fffj(R»'  V  •••>  V  k> 

l”i  t1  -  f<Ra,’  ar  V  k)T—  ’ 


where 


(10)  fR  (km.  0f,_ ,  ■  ■  ■  ,  arp ;  k)  = 


o'] ,  •  •  •  ,  a  ;  k) 


m 
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an  d 


31  (K 


k) 


(ID 


(R 


k)  - 


3a. 


P  ■ 


The  vector  (H^,  , 


or  )  lor  which 
P 


(12) 


3  lofie  ■£  3  lo«e  & 


3  logc  £ 

3qp 


^  0 


yields  the  maximum  likelihood  estimators  R  ,  q>  ,  ,  .  .  ,  a  of  the 

1  p 

parameters  R^,  cc^ ,  respectively. 

In  general,  the  system  of  equations  (12)  can  only  be  solved  by 

l\  A  A 

iterative  methods  so  Lhat  initial  estimates  R  .  cr,  .....  ct  arc 

<",o  lo  po 

needed  for  the  iteration  scheme.  Often  the  initial  estimates  are 

obtained  by  the  "least  squares"  method.  That  is,  we  minimize  the  sum 

/i  squares  Y  of  deviations  of  the  observed  success  ratios  s  /n  from 

K-  K 

their  expected  success  ratios  f(R  ,  a.  ,  •  •  •  ,  a  ;  k)  ,  with  respect  to 

<*>  1  }> 

the  pararne-.  ar  vector  (R  ,  <y,  ,  .  .  .  ,  <*  )  •  Thus,  we  have 


(13) 


3Y 

3R 


-2 


E 

k=l 


[sk/nk 


f(R 


O' 


k)]f,,  <R 


k) 


3Y 

3a. 


■-2Ec 


sk/nk 


k-1 


f(R  ,  a,  ,  •  •  • ,  a  ;  k)]f  (R  ,  a,  ,  ■  •  .  ,  a  ;  k) 

CD  j  ]*  (*•  J-  p 


(14) 
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Among  those  vector  sets  for  which 


ay  =  ay 
a». 


ay 

So 


-  o, 


we  find  the  vector  set  (R  ,  a  ,»•••»  y  )  that  is  the  least  squares 

oo(o  lo  pO 

estimator  of  (R  ,  a, ,  ■ . . ,  a  ) ■ 

co  L  p 

Iteration  schemes  are  discussed  in  more  detail  as  we  analyze  each 
proposed  model  in  the  Appendix.  We  discuss  in  particular  the  two- 
dimensional  Newton  s  Method  used  to  obtain  numerical  solutions  of  the 
maximum  likelihood  equations  in  several  of  the  models. 

Let  us  set  R  3  a  for  the  remainder  oi  this  discussion  The 

co  O 


maximum  likelihood  estimators  ,  a  ^  , 


’  °p  of  “o-  al . V 

respectively,  are  jointly  normally  distributed  when  the  samj  le  size 
m  *  22  rik  is  latSc'»  provided  tlie  following  conditions  are  satisfied 
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9  log,,  g 
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\  —  /  , 
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i ,  j  =  0 ,  1,  ....  p,  k  «=  1,  2,  N . 
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is  bounded  for  all  possible 


values  o£  aQ,  c^,  ■  ■  ■,  o^,  i,  j,  t  =  0,  1 . .  and 

at  each  stage  k. 


3  logc  X 


=  0,  i  •«  0,  1,  p.  That  is  to 


i  I 

say,  the  derivative  of  log  X  vanishes  at  its  maximum 

<v  “i>  ■■■•  V' 

Assuming  that  conditions  (l)-(4)  hold,  the  maximum  likelihood  estimators 
have  the  approximate  joint  normal  density  with  means  cyq ,  ,  .  ..,  a  and 

variance-covariance  matrix 


where, 


/d2  log  A  /d2  log  x 

■  EhH  ■ E  k*r 


&  logc  £ 


5  lo'  c 
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For  a  furLlier  discussion  of  the  large  sample  properties  of  maximum 
likelihood  estimators,  the  reader  may  refer  to  Cramer  [3,  pp .  497-506] 
and  Kendall  [5,  pp .  1-49]. 

At  this  point  we  note  that  in  some  of  the  simulation  studies  oi 
Sec.  Ill,  condition  (5)  is  violated.  This  explains  the  deviation 
from  asymptotic  normality  which  is  observed  there. 

We  now  derive  an  approximate  lOOr-perccnt  lower  confidence  limit 
for  R.  --the  predicated  reliability  at  the  k-th  stage  of  testing.  To 
accomplish  this  objective  we  need  to  obtain  an  approximate  expression 
for  Var  .  In  the  first  place,  R^  =  f  (o^ ,  ,  •••,  ’>  k)  ,  where  we 

assume  that  £(ctQ,  a^ ,  a^ ;  k)  is  at  least  twice  differentiable  in 

each  of  the  variables  (<yQ,  aj ,  a^)  .•  and  whose  second  derivatives 

are  bounded  for  all  possible  values  of  (ao ,  ,  •••,  a^)  at  each  stage 

*  — -X 

k.  We  than  may  approximate  Var  in  terms  of  ^  by  expanding  in 
a  Taylor  series  about  (aQ,  a^ ,  •••,  u^)  anc'  ignoring  terms  of  order- 
greater  than  one.  Titus, 


05) 


P 

Var  R,  =  V  f2  Var  a,  +  2  f  f  c 

k  /  .  a  cxi  r  on  a. 

i=o  1  osiejsp  1  J 
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and  Var  and  cov  (c^j  a are  the  elements  of  ^ 

Using  the  theory  developed  in  this  section,  we  develop  in  the 
Appendix  the  parameter  estimators,  examine  concavity  problems,  and 
obtain  lower  confidence  limits  for  the  four  models  described  in  this 
Appendix. 


Ill,  BEHAVIOR  QF  THE  ESTIMATING  MOPELS 

DESIRABLE  CHARACTERISTICS 

Before  examining  the  behavior  of  some  of  the  estimating  models, 
iL  will  be  helpful  to  describe  the  characterise ics  that  would  be  de¬ 
sirable  in  such  a  model.  Clearly,  we  would  like  our  model  to  come  as 
close  as  possible  to  the  "right"  answer.  A  mathematician  or  statis¬ 
tician  would  describe  this  trait  for  an  analogous  (but  not  identical) 
situation  as  requiring  minimum  variance  and  minimum  bias,  where  the 
variance  expresses  quantitatively  the  variability  oi  prediction,  and 
the  bias  expresses  the  difference  between  the  correct  answer  and  the 
average  of  a  number  of  predictions.  For  our  purposes,  the  square 
root  of  the  variance-- the  standard  deviat ion- - is  probably  more  useful, 
preserving  as  it  does  the  physical  units  of  the  original  measurement . 

Mathematical  considerations  also  suggest  that  we  should  like  to 
arrive  ct  our  estimate  by  the  method  of  maximum  likelihood  because  of 
the  favorable  large- samp] e  properties  of  such  estimates  (mentioned 
previously  in  Sec.  IT),  as  well  as  their  inherent  efficiency  in  esti¬ 
mation.  Then  too,  we  should  like  to  be  able  to  make  specific  confi¬ 
dence  s LatemenL s- - for  example,  we  are  90-percent  confident  that  the 
true  reliability  is  no  less  than  some  specified  amount;  two  elements 
contributing  to  a  direct  confidence  calculation  are  the  asymptotic 
normality  of  maximum  likelihood  estimates,  as  well  as  the  variance- 
covariance  matrix  generated  in  the  course  of  the  maximum  likelihood 
solution . 

We  would  like  our  estimate  to  be  relatively  insensitive  to  three 
environmental  features  over  which  wc  may  have  little,  or  no,  control. 


HWmtitr— . . * . 
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First,  by  necessity,  th“  decision  to  allocate  a  particular  trial  to 
one  stage  of  testing  or  another  may  be  quite  arbitrary  (the  configura¬ 
tion  may  change  very  slightly  from  one  trial  to  the  next,  obviating  any 
attempt  to  populate  a  given  stage  with  homogeneous  items).  We  should 
thus  like  to  have  arbitrary  grouping  be  of  lit  Lie  consequence  to  the. 
resulting  estimates.  Second,  the  form  of  the  actual  underlying  growth 
curve  should  have  as  little  effecL  as  practical.  In  other  words,  ihu 
estimate  should  not  be  sensitive  to  whether  a  given  level  of  reliability 
wa  .  reached  by  vigorous  early  growth,  followed  by  a  tapering  off  to  a 
virtually  constant  value,  or  by  a  slow,  sustained  growth  process. 

Third,  the  estimating  model  should  be  able  to  ao  its  job  whether  the 
predicted  reliability  is  in  the  region  of  0.5,  or  ru.ar  1.0,  or  anywhere 
else,  in  the  reliability  spectrum. 

Since  estimates  will  be  reached  through  extensive  computations,  it. 
is  desirable  that  these  be  reasonably  compatible  with  modern  '.imputing 
methods,  that  is,  with  digital  computation.  Thus,  a  good  model  should 
result  In.  a  computing  algorithm,  or  routine,  thai  is  unlikely  to  iead 
to  difficulties  such  as  spurious  roots,  divisions  by  zero,  logarithms 
of  zero,  and  other  s.tmbliug  blocks,  or  to  result  in  instabilities,  or 
divergences,  if  iterative  methods  must  be  used.  Likewise,  convergence 
to  the  proper  answer  should  b  reasonably  prompt,  with  no  excessive 
"hunting.” 

Finally,  the  estimating  model  should  bear  some  strong  resemblance 
to  physical  reality,  and  must  be  conipat ible  with  the  mathematical 
in  lerp  re  La  L  ion  of  reliability.  Foi  examp'e,  the  parameters  of  the 
model  (L.u.,  the  quantities  for  which  we  will  make  numerical  estimates) 
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might  he  such  things  as  the  Lni.li.al  reliability,  ultimate  rel  i.abLl  L  ty  , 
initial  growth  rate,  etc.  ,  and  any  numerical  quantity  denoting  relia¬ 
bility  must  Laice  on  values  neither  lees  than  zero,  nor  more  than  one. 

^VALUATION  HETilODOLOCY 

Few  of  the  foregoing  qualities  can  be  implemented  by  straightforward 
analytical  efforts,  because  of  the  excessive  complexities  involved.  For 
this  reason  we  chose  to  study  the  behavior  and  "optimization"  of  the 
reliability  growth  models  through  MonLe  Carlo,  or  simulation  methods. 

The  most-used  procedure  was  to  simulate,  on  a  digital  computer,  a  test 
program  of  7  2  fiats  with  a  given  underlying  growth  charac  Leris  tic  . 
usually  exponential.  Ir.  most  cases,  the  first  l?.  trial  results  were 
then  combined  to  form  the  first  "stage,"  the  next  12  to  form  the.  second 
"stage,"  and  so  on,  giving  six  groups  of  12  trials  apiece,  with  the 
reSuLcc  punched  on  a  smgle  IBM  card.  This  process  was  repeated,  either 
99  times  for  rough  compari  .iorif ,  or  999  Limes  for  mote  detailed  ones. 

These  data  cord  deebs  were  then  analyzed  with  different  reliability 
growth  modeis,  and  the  99  (or  999)  resulting  predictions  of  reliability 
in  the  next  (i.e.,  seventh)  stage  of  testing  were  used  as  indicators 
of  estimating  ubtlt'y.  The  usual  measures  of  quality  were  the  standard 
deviation  of  the  estimates,  and  the  bias,  the  difference  between  the 
average  of  Lite  estimates  and  the  "correct"  answer  from  the  underlying 
growth  characteristic  used  lo  simulate  Lhe  data.  Another  thing  considered 
was  the  distribution  of  the  estimates,  the  Normal  or  Gaussian  being 
uxpee  Leu  . 

X'h?  m,st  substantial  devl-  tons  from  the  foregoing  procedure  were 
made  wucr.  studying  lhe  effect  of  grouping,  or  dividing  the  72  trials 


-le- 


intc  stages.  In  tills  casfij  the  identical  72  individual  trials  were 
divided  into  four  groups  of  18  trials  apiece,  sir.  groups  of  12  trials, 
and  eight  groups  of  9  trials,  as  shown  in  Table  1. 

Table  1 

EFFECT  OF  VARIED  GROUPING  IN  999  PROGRAMS  OF  72  TRIADS 


Trials 
Per  Group 


Predic  tion 


an  Estimate 

Actual 

Bias 

0.771 

0.838 

-0.067 

0.793 

0.830 

-C.037 

0.807 

0.825 

-0.018 

0.808 

0.812 

-0.004 

The  first  three  lines  ot  Table  1  show:  the  number  of  groups  and 
trials  per  group  jusL  mentioned;  the  actual  reliability  and  the  average 
of  999  predictions  for  the  N+lst  stage;  the  difference  (bias)  between 

the  latter  two,  and  the  standard  deviation  of  the  999  predictions  with 

*  fl-k3/A 

the  generalised  model,  -  aF(k),  and  with  F(k)  =  c  '  The 

fourth  line  of  the  table  shows  a  comparison  with  analysis  by  the  cx- 

-  Ak 

ponential  model,  R,  -  1  -  etc  *■  ,  where  each  individual  trial  consti¬ 
tutes  a  stage  of  testing.  Because  of  piactical  limitations  on  the 
function  F(k),  Lite  generalized  model  cannot:  dc  extended  to  analyze 
more  than  about  12  stages.  The  solution  for  the  estimates  of  the 
parameters  of  the  exponential  model  suffers  convergence  difficulties 
with  more  than  one  trial  per  stage. 


For  purposes  cf  facilitating  recognition.  Model  1  will  henceforth 
be  referred  Lo  as  the  "generalized"  model,  alluding  to  the  wide  range 
of  choices  for  F(k),  while  Model  2  will  be  called  Llic  "exponential" 
model  for  what  should  be  an  obvious  reason. 
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Two  major  conclusions  can  be  drawn  from  the  table:  (1)  arbitrary 
grouping  makes  relatively  little  difference  in  the  average  prediction, 
though  more  numerous  groups  wiLh  fewer  trials  give  slightly  higher 
(and  in  this  case  less  biased)  l-esults;  and  (2)  more  numerous  groups 
result  in  a  mode  L' a  tuly  higher  variability  of  predict  inn.  Tills  raises 

the  question  of  how  large  a  standard  deviation  one  should  reasonably 
expect  under  the  circumstances.  While  the  answer  to  that  is  not  easily 
found,  we  can  get  at  least  some  idea  from  a  somewhat,  different  case  for 
which  the  exait  answer  is  well  known. 

If,  Instead  of  "/2  trials  with  progressively  increasing  reliability, 
we  had  n  triaLs  with  a  constant:  reliability  of  p  (called  Bernoulli  or 
Binomial  trials) ,  the  standard  deviation  of  p,  the  estimate  of  p,  is 


a* 


1> 


which  for  our  case  (p  =  0.83,  n  =  72)  gives  a  =  0.04h.  Clearly,  we 
arc  not  in  as  favorable  a  position  as  this  since  the  reliability  is 


vu  ryrng , 
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near  0.044,  as  for  example  the  one  for  N  -  4  groups.  Later,  we  will 
see  further  reason  for  caution  in  this  regard. 


Another  way  of  looking  at  Lite  effect  of  grouping  is  to  examine  a 
particular  series  of  72  trials  (one  of  the  999  examined  previously),  as 
lit  Table  2.  In  this  instance  (which  is  not  necessarily  representative) 


'k 

The  standard  deviation  for  72  trials  is  a  consequence  of  the  model 
used  for  analysis  (the  exponential)  rather  than  the  grouping,  or  lack 
of  grouping.  The  exponential  model  used  to  analyze  data  grouped  in 
six  stages  gave  nearly  identical  results  to  the  72  si. ages,  indicating 
complex  insensitivity,  but  with  the  computational  problems  noted  earlier. 
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Table  2 

EFFECT  OF  VARIED  GROUPINC  ON  ONE  PROGRAM  OF  72  TRIALS 


Number 
of  Groups 

Re i iabi 1 ity 

Estimates 

Next  Stage 

Nh-i 

Ultimate, 

R 

CD 

2 

0.7122 

1.0+ 

*3 

0.7480 

1.0+ 

4 

0.7820 

0.995 

6 

0.7748 

0.8915 

8 

0.7481 

0.8114 

9 

0.7566 

0.8090 

12 

0. 7r78 

0.7393 

72 

0.780 

-- 

increased  numbers  o£  groups  (more  than  2  and  3*)  result  in  generally 
lower  esimtates,  but  once  again  only  slightly  so.  In  summary,  the  simu¬ 
lations  show  that  arbitrary  grouping  of  trials  has  relatively  little 
cffc.ct  on  the  resulting  predictions.  As  we  will  see  a  little  laLer, 
systematic  grouping  can  have  somewhat  larger  and  worthwhile  effects. 

We  have  just  seen  an  example  in  which  the  maximum  likelihood  esti¬ 
mate  given  by  the  generalized  model  for  R  ,  the  ultimate  reliability, 
was  larger  than  1.0,  a  result  that  is  incompatible  with  the  mathe¬ 
matical  restrictions.  Fortunately,  Lbc  likelihood  function  for  this 
model  is  always  concave  downward,  so  that  the  so-called  "constrained 
maximum,"  where  R  is  required  to  be  less  than  or  equal  Lo  1.0,  must 
occur  on  the  boundary  whenever  the  unconstrained  case  gives  R  above 
1,0.  Knowing  this,  We  can  simply  sei  R  —  1.0  in  these  cases,  and 

<X> 

recalculate  the  maximum  likelihood  estimate  a- 

Figure  1  illustrates  the  three  things  thaL  happen  when  this  "ii.mi- 
Laling"  process  is  implemented.  The  plot  shows  the  ranked  values  from 

The  exclusion  of  the  two  and  three-group  cases  was  made  because 
these  both  resulted  in  estimates  for  R^,  the  ultimate  reliability,  in 
excess  oi  1.0,  a  topic  that  will  be  addressed  next. 
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the  estimate  o£  R^+^)  values  from  99  test  programs  of  72  trials  each, 

where  both  the  underlying  growth  and  the  analysis  followed  the  general" 

2 

ized  growth  model  with  F(k)  =  .  (Tliis  type  of  plot,  on  normal- 

probability  paper,  is  frequently  used  to  show  the  relationship  of  test 
results  to  the  Normal  distribution.)  The  solid  dots  are  the  ranked 
predictions  where  R  is  not  restricted,  and  show  the  expected  normality 
(the  straight  line) .  The  open  dots  show  the  results  where  limiting  lias 
been  implemented,  indicating  that:  (1)  the  results  are  no  longer  no.mal 
thus  complicating  confidence  calculations;  (2)  the  average  result  has 
been  biased  downward  by  reducing  tint  high  estimate1:  while  not  affecting 
the  low  ones;  and  (3)  the  standard  deviation,  a  measure  of  the  vari¬ 
ability  of  prediction,  has  been  reduced. 

The  first  of  these  effects  is  clearly  detrimental,  since  it  counter 
acts  one  of  the  desirabLe  features  of  maximum  likelihood  estimation. 

The  second  is  beneficial  in  this  instance;  without  limiting,  the 
estimates  are  biased  high.  However  this  is  not  always  the  case,  as 
we  shall  see.  The  Ihlrd  effecL,  Lhc*  reduction  in  standard  deviation. 

Is  generally  desirable,  but  we  note  that  the  effect  occurs  entirely 
because  of  the  reduced  estimates  at  the  higher  end  of  the  spectrum, 
thus  giving  rise  to  the  other  two  noted  features.  It  should  be  evident 

that  when  the  estimate  of  K  is  above  1.0,  one  might  be  well-advised 

00 

to  choose  a  different  l'(k)  or  a  different  model,  rather  than  to  lollow 

■A 

the  procedure  mentioned  earlier. 

One  should  not  get  the  impression  that  the  model  will  be  chosen 
to  fit  the  data  or  chosen  after  the  data  have  been  examined.  The  form 
of  the  model  is  not  changed.  Vie  change  only  one  oi  the  parameters  so 
the  model  meets  physical  constraints,  I.e.,  0  <  Koj  S  1.0.  Later  in 
this  section,  one  such  change  will  be  discussed  in  detail. 


y  scale 
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Before  making  some  direct  comparisons  between  estimating  modeLs, 
one  additional  refinement  applicable  to  the  generalized  model  will  be 
discussed.  In  a  reliability  research  study  for  NASA,  Barlow  and 
Schcuer  [1]  suggested  a  method  for  o bta Lning  maximum  likelihood  esti¬ 
mates  for  past  or  current  (but  not  future)  stages  of  testing  in  the 
presence  of  reliability  growth.  One  feature  of  their  procedure  is  a 
regrouping  process,  whereby  in  a  series  of  testing  stages,  adjacent 
stages  are  combined  wherever  the  success  ratio  (successes  divided  by 
trials)  in  the  later  stage  is  lower  than  In  the  earlier  stage.  The 
process  is  continued  until  all  such  "reversals"  are  eliminated.  Tills 
process  is  of  substantial  benefit  to  the  qualiLy  of  estimates  made 
with  the  generalized  model. 

Figure  2  is  a  bar  chart  intended  to  illustrate  both  the  incidence 
and  che  size  of  the  benefits  achieved  when  this  process  was  applied 
to  the  data  from  99  test  programs  with  underlying  hyperbolic  reliability 
growth  before  using  the  generalized  model.  In  each  case,  the  data 
originally  consisted  o£  six  stages,  each  having  12  trials.  Several 
of  tile  99  programs  had  no  reversals,  and  thus  still  had  six  stages 
after  processing.  The  leftmost  bar  in  Fig.  2  shows  that  the  standard 
deviation  remained  unchanged  for  these  programs.  The  next  bar  shows 
that  those  programs  that  had  one  reversal  enjoyed  a  slight  reduction 
in  <7,  from  0.039  to  0.036  (the  shaded  area);  the  remainder  of  the 
charL  shows  how  programs  with  two,  three,  and  lour  reversals  fared. 
Clearly,  before  the  process  was  applied,  the  programs  with  more  rever¬ 
sals  had  a  higher  variability  of  prediction  than  those  with  fewer  or 
no  reversals,  but  regrouping  according  to  the  Bar low-Scbouer  procedure 


removed  this  disadvantage.  Although  the  reasons  for  expecting  benefits 
by  applying  this  regrouping  process  are  intuitive  in  this  case,  and 
based  on  generally  empirical  observations  concerning  the  behavior  of 
maximum  likelihood  estimators,  the  fact  remains  that  the  process  usually 
improves  estimates  where  reversals  of  success  ratio  are  present. 

Since  Lha  purpose  of  the  Ba rlow -Scheucv  effort  is  reliability 
assessment  in  the  presence  of  reliability  growth,  one  might  reasonably 
ask  why  we  did  not  use  their  general  method  rather  than  just  one  feature 
of  it.  There  are  basically  four  reasons  why  we  chose  to  take  a  new 
approach . 

1.  Application  of  the  trinomial  model  requires  ass ignmen i  of 
failures  to  "inherent"  or  "assignable  cause"  categories, 
something  which  is  often  simply  impossible. 

2.  Stages  should  either  he  homogeneous  or  end  with  an  assign¬ 
able  cause  failure,  both  hard  to  satisfy. 

3.  There  is  no  way  to  extrapolate  to  the  N+lst  stage. 

4.  The  confidence  bound  is  inadequate,  penalized  too  much 
by  early  test  results. 

Of  these,  only  the  latter  two  are  critical,  since  relaxation  of  the 
first  two  is  possible  within  the  framework  of  the  methodology.  Further, 
the  third  may  be  less  important  late  in  a  test  program,  provided  growth 
has  substantially  abated. 

The  reader  should  now  have  sufficient  background  to  appreciate 
the  performance  differences  between  the  two  principal  candidate  models, 
the  exponential  and  the  generalized  hyperbolic.  Those  models  were 
used  to  analyze  data  generated  from  three  substantially  different 
growth  curves,  each  having  a  reliability  in  Lhe  seventh  (i.e.,  next) 
stage  of  0.8  to  0.85.  The  1  i.rst  growth  curve  was  exponential  with 
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slow  but  persistent  growth,  the  second  a  modified  hyperbolic,  and  the 
third  a  hyperbolic  growth,  vigorous,  but  short- lived .  Once  again, 
six  stages  of  12  trials  apiece  were  used  for  the  generali_ed  model 
with  limiting  of  R  and  regrouping  to  eliminate  reversals;  Lhe  ungrouped 

Oj 

data  (72  stages)  for  the  same  trials  were  used  with  the  exponential 
model.  The  form  function ,  E(k) ,  used  with  the  generalized  model  was 
e''  *'  ,  which  is  a  desirable  compromise  between  functions  giving 

excessive  bras  and  those  giving  excessive  variability.  Table  3  shows 
the  results,  giving  comparisons  of  bias  and  standard  deviation  for 
the  two  models  applied  to  the  three  different  growth  curves. 


Table  3 

EXPONENTIAL  AND  GENERALIZED  MODELS  COMPARED 
(Estimates  Based  on  72  Trials) 


Underlying  Growth 
Characteristic 

Bias 

Deviation 

1 

Exponent ial 

Generalized 

Exponential 

Generalized 

R  =■  1.0  -  <ye'Bk 

0 

-0.03 

0.0603 

0.0403 

K 

(0 . 0726) a 

(0-0494)“ 

\  *  K~  ’  "  T-k 

+0.013 

-0.015 

0-0648 

0.0445 

R,  =  R  -  ar/k 
k 

+004 

-0.01 

0.0574 

0.0401 

max.  difference 

between  cases 

0.04 

0.02 

0.0074 

0.0044 

a999  programs . 


The  generalized  model  shows  advantages  in  three  important  quali¬ 
ties.  The  standard  deviation,  a  direct  measure  of  the  variability  of 
the  estimate,  is  consistently  lower  than  that  for  the  exponential  model, 
regardless  of  tic  type  of  data  being  analyzed.  Also,  the  sensitivity 


-27- 


to  the  type  of  data  is  lower,  as  indicated  by  the  final  line  of  entries. 
(Toe  spread  of  deviations  is  more  than  proportionately  larger  for  the 
exponential.)  Finally,  the  net  bias  is  routinely  negative  (i.e.,  low, 
or  conservative  estimetes)  by  contrast  with  the  potentially  large  posi¬ 
tive  bias  of  the  exponential  model  when  used  to  analyze  vigorous  growth 
da  la  . 

IWo  potential  drawbacks  of  the  generalized  model  are  also  evident. 
For  exponentially  generated  data,  it  is  biased  where  the  exponential 
model  is  not,  and  there  are  valid  theoretical  reasons  (though  r\o  demon¬ 
strable  evidence)  to  expect  exponential  data  to  be  more  common  than 
other  kinds.  ALso,  the  standard  deviation  is  once  again  suspiciously 
low,  though  not  as  much  so  as  the  figures  would  indicate.  The  99  pro¬ 
gram  runs  used  here  for  analysis  had  somewhat  less  variability  (in  terns 
of  the  number  of  successes  in  each  stage)  than  would  normally  be  ex¬ 
pected.  The  figures  in  parentheses  are  for  a  999  program  set  of  data, 
which  were  more  representative,  and  which  confirm  the  superiority  of 
the  generalised  model  at  a  somewhat  more  reaListic  level  of  standard 
deviation. 

A  graphical  representation  offers  another  means  of  comparison. 
Figure  3  shows  the  999  predictions  from  both  the  exponential  and 
generalized  models  plotted  according  to  mean  rankings  on  normal  proba¬ 
bility  paper.  The  lower  standard  deviation  (smaller  slope)  for  the 
generalized  model  is  quite  evident.  The  bias  of  the  generalized  model 
stands  out  even  more  clearly;  in  approximately  nine  cases  out  of  ten, 
the  generalized  model  gives  a  numerical  value  below  the  exponential 


generalized  and  exponential  model'  for  999  cases 
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model  .  and  in  only  one  case  of  the  999  cud  the  value  ewe  red  U  90.  Tic 
deviation  from  a  normal  distribution  is  also  quite  evident. 

A  specific  (and  quite  unrepresentative)  example  from  the  999 
program  run  may  serve  to  dramatize  tie  concern  with  a  "realistic" 
standard  deviation,  and  to  pinpoi  .i  th  ~.  p.-tent  i.ii  ly  serious  short¬ 
coming  of  this  particular  version  of  the  generalized  model.  The  solid 
Line  in  Fig.  4  shows  the  underlying  growth  characteristic  (reliability 
versus  stage  of  testing)  used  to  simulate  the  999  test  programs.  The 
solid  dots  show  what  was  probably  the  most  unusual  of  the  999  rcsulLs, 
with  experience  in  the  first  three  stages  considerably  below  the  ex¬ 
pected  success  ratio,  and  i.n  the  last  three  sLages  considerably  above 
the  expected.  Tie  dashed  line  shows  how  the  generalized  model  fits 
a  growth  curve  lo  these,  data,  giving  a  prediction  for  stage  7  which 
is  quite  close  to  the  "correct"  answer.  The  dolled  line  shews  how 
the  exponential  model  interprets  these  Jcta,  giving  s  much  higher 
prediction.  In  spite  of  the  fact  that  the  prediction  the  exponential 
model  made  is  substantially  further  from  the  true  reliability,  It 
should  be  evident  that  this  higher  prediction  is  eminently  more  reason 
able  in  view  of  the.  experience  data  itself.  In  fact  ,  the  generalized 
model  with  the  k) /4  function  is  simply  unable  to  cope  with  growth 

that,  approaches  a  reliability  of  1.0  at  anything  more  than  a  slow 

pace.  II  a  function  with  somewhat  more  downward  concavity  is  used 
1  -  k 

(for  example,  e  ),  the  ci  te -.L  is  less  pronounced. 

y? 

In  each  case,  a  bcLa  distribution  with  the  appropriate  mean  and 
variance  represented  the  distribution  ol  results  jar  bet.tci  then  the 
normal.  The  beta  Is  especially  good  tor  the  exponential  results,  and 
the  confidence  interval  for  that  model  war.  thus  calculated  accordingly 
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The  shortcomings  of  the  general ized  model  In  analyzing  data  from 
a  population  whose  reliability  is  approaching  1.0  led  to  the  search 
for  another  model  to  deal  with  this  important  case.  To  avoid  (at  least 
for  the  time  being)  introducing  a  third  estimated  parameter,  the 
generalized  model  was  modified  In  a  simple  way  that  would  achieve 
somewhat  the  same  purpose  without  much  extra  analytical  complication. 
Briefly,  the  reliability  growth  equation  is  as  follows: 


*k 


( i  -k)  /:i 


where  the  coefficient  N  is  initially  set  to  an  Integer  between  4  and 

8  (usually  6  in  our  examinations)  ,  and  both  R  and  a  are  estimated. 

00 

«** 

If  R  turns  out  to  be  1.0  or  less,  the  estimate  is  accepted.  Other- 

CD 

wise,  N  is  reduced  by  1.0,  and  the  process  is  repeated  until  R^  drops 

A 

to  1.0  or  below:  however,  if  R  remains-  in  excess  of  1.0  when  N  «*  1  , 

00 

then  the  previously  described  limiting  process  is  introduced,  such 
that 

*Tc  1  ~ 

is  used  to  solve  for  the  single  parameter  a- 

The  characteristics  of  this  adaptive  model  can  be  appreciated 
best  by  comparing  them  with  the  exponential  model,  as  in  Fig.  3,  which 
shows  the  ranked  prediction::  foi"  both  models  when  used  to  analyze  the 
same  999  test  programs  as  before.  This  time  the  superiority  of  the 
adaptive  model  over  the  exponential  model  Is  achieved  without  the 
drawbacks  (negative  bias  and  extreme  ncnnormal ity)  that  character ize 
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the  generalized  model.  This  time  the  result  for  each  model  is  closely 
approximated  by  a  beta  distribution  with  matching  first  and  second 
moments . 

In  spite  of  an  apparent  superiority  to  the  exponential  model,  the 
adaptive  model  does  have  shortcomings.  It  shares  with  the  generalized 
model  the  inability  to  make  a  believable  estimate  for  the  case  previously 
described  in  Fig.  4.  (Indeed  no  other  growth  function  was  found  that 
would  make  an  estimate  as  high  as  that  for  the  exponential  model.) 

Since  that  case  is  anything  but  typical,  it  is  probably  more  important 
to  note  that  the  adaptive  model  also  shares  a  defect  of  the  exponential; 
it  overestimates  reliabilities  wherever  Lhe  asymptotic  reliability  is 
substantially  below  L.O.  Thus  none  of  the  models  described  here  are 
universally  applicable.  One  must  have  some  notion  regarding  the  asymp¬ 
totic  reliability  if  bias  is  co  be  avoided  (Table  4). 

Table  4 


EFFECT  OF  ASYMPTOTIC  RELIABILITY  ON  BIAS  FOR  THREE  MODELS 


Asymptotic 
Rel  iabi  1  L  t.y 

Genera l i zed 

Adaptive 

Exponen  t la l 

R  -  L.O 

CO 

Biased  low 

No  bias 

No  bias 

K  -  0.9 

00 

No  bias 

Slightly  high 

Slightly  high 

R  =0.7 

CO 

No  bias 

Very  high 

Very  high 

If  we  are  either  unwilling  or  unable  to  decide  whether  the  asymp¬ 
totic  reliability  is  near  1.0,  then  it  becomes  necessary  to  introduce 
a  three-parameter  model,  which  estimates  the  three  physical  character¬ 
istics:  ultimate  (asymptotic)  reliability,  starting  reliability  (alter¬ 

natively,  arnJun-  of  growth),  and  a  measure  of  the  rate  oi  growth- 


Developing  such  a  model,  however,  is  much  more  complex  than  developing 
the  two-parameter  models,  and  will  not  be  attempted  here. 

SOME  ADDITIONAL  COMPARISONS 

Since  nonparainelric  methods  have  a  strong  appeal  lor  use  in  relia¬ 
bility  assessment,  our  description  of  the  proposed  parametric  (i.e., 
assuming  an  underlying  model)  methods  would  be  incomplete  withouL  a 
comparison  to  some  pertinent  competitors  from  the  nonpa ramc tr Lc  field. 
The  two  of  these  to  be  used  are  the  method  of  Bar lew  and  Scheuer  [l], 
and  an  extension  (described  at  the  end  of  Sec.  II)  to  a  method  suggest¬ 
ed  by  fox  [A],  involving  Hayes'  theorem.  The  latter  method  was  eval¬ 
uated  for  the  conventional  situation  where  no  growLh  is  assumed,  and 
for  a  second  case  where  a  rather  accurate  representation  of  the  growth 
between  stages  was  superimposed  on  the  process, 

T'abLe  b  shows  the  results  of  the  calculations  applied  to  the  series 
of  99  cases  (with  exponential  growth) ,  which  we  previously  noted  had 
somewhat  less  than  the  expected  variance  in  outcomes.  The  table  shows 
that  tor  prediction,  the  adapLive  model  gives  lower  variability  Lhan 
any  of  the  nonparametr ic  methods  used,  and  less  bias  Lhan  either  of 
the  Bayes  approaches.  While  the  liarlow-Scheuer  resu  L  t  is  actually 
closer  on  the  average  (albeit  more  variable),  it  should  be  noted  That 
this  is  act  ually  the  same  result  usee  to  assess  Stage  N  (the  Barlow  - 
Scheuer  method  does  not  include  prediction)  and  is  a  lucky  accident. 

T.n  assessing  reliability  oi  the  most  recent  stage,  the  adaptive 
method  shows  a  clear  superiority  in  both  measures.  The  Hat  low- Scheuer 
results  arc  biased  high,  largely  the  result  of  tiic  substantial  number 
of  cases  where  11  or  12  successes  occurred  m  the  sixth  stage,  giving 
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TabLe  5 


A  COMPARISON  OF  EVALUATIONS  FOR  99  CASES 


Me  Lhod 

Pro diction 
of  R(N+L) 

Assessment 
of  R(N) 

Lower  957,  - 
Bound  for  R(N) 

Condi t  Lons 

u 

o 

U 

a 

U 

— 

Q 

a 

X 

Adaptive 

0.8353 

0.0513 

0.7898 

0.0595 

0.6635 

0.0690 

3 

Normal  Dlst . 

0.6514 

0.0667 

1 

Beta  hist. 

Bar low-Scheuer 

0.8276 

0.0776 

0.8276 

0.0776 

0.529 

0.0537 

0 

Bayes 

0.8399 

0. 1117 

0.7763 

0.1036 

0.6905 

0.0869 

11 

With  Growth 

0.7974 

0. 1126 

0.7363 

0.1107 

0.6565 

0.0948 

8 

Without  Growth 

"Correc  t" 

0.83 

_ 1 

0.789 

-- 

-- 

5 

aNumbcr  of  lower  bounds  exceeding  actual  value.  Note  that  while  the  lower 
bounds  for  the  adaptive  model  have  no  rigid  mathemat i ca 1  validity,  they  still 
give  worthwhile,  if  conservative,  results.  This  conservatism  is  typical  of 
both  the  generalized  and  exponential  models  as  well,  indicating  that  the  esti¬ 
mated  variance  is  substantially  larger  than  the  actual  in  the  vast  majority 
of  cases  examined. 

estimates  of  0.9167  or  1.0.  The  Bayes  method  is  biased  very  low  if  no 
growth  is  assumed,  and  in  still  slightly  iuw  when  a  virtually  exact 
representat-  Ion  of  the  growth  is  supplied.  The  adaptive  method  gives 
unbiased  results,  on  the  average,  and  with  much  less  variability. 

Comparing  the  lower  bounds  lor  the  90-percent  confidence  interval 
shows  how  the  adaptive  method  avoids  the  excesses  ol  the  other  two. 

The  Bariow-Scheuer  results  arc  excessively  conservative.  None  ol  the 
99  estimates  were  above  the  actual,  where  5  inigh'.  normally  he  antici¬ 
pated  lor  a  valid  lower  bound.  (In  a  later  run  cf  999  cases,  there 
were  still  none,  where  50  should  be  expected.)  The  Bayes  results  at e 
Loo  optimistic,  even  though  the  mean  l'esult  without  growth  is  almost 
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identical  to  the  adaptive  results.  The  adaptive  method  gives  results 
that  are  only  slightly  conservative,  and  should  thus  logically  be 
preferred  over  the  other  two. 
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IV.  CONCLUSIONS 


On  the  basis  of  substantial  and  independent  previous  examination 
of  the  general  topic  (primarily  Refs.  7,  8,  9,  10,  and  11)  the  followin 
conclusions  can  be  drawn: 

1.  Progressive  growth  in  the  reliability  of  certain  types  of 
large  weapon  systems  appears  to  be  charac ter is t ic  of  their 
development . 

2.  Although  parametric  models  require  more  assumptions  con¬ 
cerning  the  reliability  growth  of  a  system,  physical  and 
engineering  considerations  often  provide  empirical  and 
intuitive  justification  for  a  characteristic  model. 

Building  on  this  foundation,  the  more  recent  research  reported 


here  permits  the  following  extensions: 

3.  A  simple  parametric  growth  model  appears  to  have  advantages 
over  other  available  approaches  for  assessing  reliability. 

A.  Not  all  of  the  parametric  models  considered  are  useful,  or 
even  feasible,  because  of  mathematical  difficulties  re¬ 
stricting  the  choice  of  parametric  form. 

5.  Parametric  growth  modeling,  In  general,  permits  extrapo¬ 
lation  of  previous  results  to  predict  near- term  future 
reliabilities.  In  addition,  the  large  sample  normality 
properties  of  maximum  likelihood  estimation  yield  a  simple 
but  effective  method  of  calculating  lower  confidence 
bounds  on  reliability  for  any  stage,  past,  present,  or 

fu  ture . 

6.  A  given  parametric  model  may  yield,  in  some  cases,  maximum 
likelihood  estimates  that  lie  outside  the  allowable  range 
for  determining  probabilities.  Under  these  circumstances 
an  "adaptive"  model  lias  been  developed  that  yields  mathe¬ 
matically  as  well  as  physically  reasonable  results. 

7.  The  models  suggested  and  studied  here  appear  to  be  rela¬ 
tively  insensitive  to  several  extraneous  and  usually 
uncontrolled  factors  such  as  (1)  grouping  into  stages,  and 
(2)  the  actual  form  of  the  underlying  growth  characteristic. 

8-  The  numerical  methods  of  solution  used  here  are  iterative, 
but  converge  rapidly  to  the  appropriate  solution.  These 
methods  are  easi  ly  implemented  on  modern  digital  gf  — outers . 
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9.  Suitable  models  are  available  not  only  for  data  that  yield 
reliabilities  approaching  1.0,  but  also  for  data  that  yield 
reliabilities  converging  to  values  considerably  less  than 
unity.  The  differences  appear  to  be  small  (by  what  seem 
to  be  reasonable  standards)  but  probably  deserving 
of  attention. 
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Append  ix 

ANALYSIS  OF  RELIABILITY  GROWTH  MODELS 


MODEL  1 


This  model  is  covered  extensively  in  Lloyd  and  Lipow  [6]  for  the 
case  F(k)  =  1/k.  Wc  treat  the  more  general  case  here  in  which 


(16) 


I(R  ,  a,  k)  *=  R  -  o-K(k). 


The  function 


(17)  log 


5e  £  =2  {l0S-(s,k)+  8k  loSe  [ 
k=l  ^  k 


p.  -  dW] 


+  (\  -  sk)  logc  fl  -  Rro  +  0fE(k)  ] 


is  concave  in  ( y.  R  ),  since 
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la  negative  definite  for  N  >  3.  U  K  D  1,  we  have  bu  one  stage,  and 
reliability  giowib  uamut  be  uaacutsed. 

Thus,  tlie  maximum  likelihood  estimators  in  Lbc  region  0  <  al'  (k)  < 
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A  third  and  more  general  approach  of  solving  (22)  and  (23)  La 
an  application  of  Newton's  method  In  two  dimensions.  For  the  sake  of 
compl e Lcnesf  we  now  present  this  method  in  general  terms. 

Suppose  Lhat  F(x,  y)  and  0(x,  y)  are,  at  least  once,  differentiable 
functions  in  the  variables  x  and  y.  We  then  wi,--h  to  solve,  iteratively, 
the  equations 


F (x  ,  y)  -  0 


G(x,  y)  -  <). 


Uld  t  tiii'j  y  tit  flitt  tm.  tu-wn 

n  th  iteration  lu  L'.ie  Solutlcn  1.3  tj.'V'Si;  as 


' : ! i'  <j  ’» vi  r»  *  Cy  l  S ! C"  o I  u  t  i  qi;  .  i.  ! i i «  t- H c.* 
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Thc  second  partial  derivatives  of  log  f  with  respect  to  R  and  a 

C  CD 

are  given  by  (18),  (19).  and  (70) 

Since  and  a  arc  approximately  disLribuLed  normally  with  means 
R^  and  a>  respectively,  and  the  variance-covariance  matrix  given 

previously,  an  approx i.ma te  lOOT-perccnt  lower  confidence  limit  for 
R  (the  predicted  reliability  at  the  k-th  stage  of  testing)  is  given 


06)  1  -  ^  - 


Z ,  VVar  R. 


*  \  -  Zl_7  y  Var  +  !•'  (k)  Var  y  -  21' (k)  cov  (fi^,  o')  , 


where  Var  R  ,  Var  a  and  cov  (R  ,  a)  are  the  elements  from  the  matrix 

CO  tt> 

.  and  is  obtained  from 


-J~2r t  ./ 


lT  'z2/2  d  1 

e  dx  =  1  -  t  . 


Since  LlieoreLical  values  will  not  be  available  in  practice,  the  maximum 
like!  ihood  estimates  k  and  a  ,n  "hst  i  luted  in  these  equations  Lo 
obtain  numerical  results,  which  appear  in  See.  III. 

As  ircquenlly  happens  with  data  analyses,  (7.1)  may  yield  a  maximum 
likelihood  estimate  of  R  that  is  greater  than  unity.  If  this  should 
occur,  R  is  set  equ  l  to  one  and  Llie  function  Lo  be  maximized  is: 


£  ■  k?,  (»*)['  - " (k>] k  [ul  (k)] k  k- 
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ii 

-°8e  X  =  const,  +  y  ]  sk  logc  [1  -  oF(k)]  +  E  <\  -  V  loge  ff> 


k=l 


k=l 


08) 


5  los  X 


s  F<k) 


(nL.  -  SL.) 


da 


c  -  X  '  k  '  '  t  i  k  k 

IT  -  al‘(k>]  + 


V'  -_k_l 

a 


k=l 


k=l 


The  maximum  likelihood  estimate  is  that  a  such  that 


(39) 


V  skr(k)  (nk  ~  sk} 

Zj  [1  -  Q'i’(k)]  Z  j  a 
k=l  k=l 


To  i  erate  on  a  solution  for  a,  wc  let  he  the  n-th  iteration; 
then  using  Newton's  method,  wc'  find 


(40) 


°Ul  °  °h  + 


k  v  ,  v  3k1(k)  j 

k  S  '°k  ~  v  '  k  h  -  »„FW3) 

l  N  N  £,!'2(k)  ) 

(“k  -  pk)  +  iz  - — — — 

(cr  k=l  k  k  k-l  r.1  -  a  r(k)  y) 


Moni'.T.  2 

This  model  is  utilized  in  Reliability  Growth  of  U.S.  Rockets  (U) 
[9];  the  model  is  not  classified.  In  this  ease. 


(41) 


i(Kro,  a1 ,  a2,  k)  =  1  -  ofj '■ 


-a2k 
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Thal  is,  R  is  assumed  Lo  be  unity  in  this  case  (which,  depending 
upon  the  particular  application,  may  or  may  not  be  a  reasonable 
assumption).  The  likelihood  function  is  then 


It  is  not  difficult  to  show  that  log^  £  is  a  concave  function  in 

and  ,  insuring  that  the  maximum  is  unique.  We  ioliow  the  approach 

a2 

in  [9j,  assuming  the  maximum  occurs  in  the  region  C  <  <  e  .  We 

now  solve  the  equations 


(43) 


3  l0ge  X 


3a, 


"k(l  ~  V'nh  ~  v'"2 )  _ 

fe  *,(,  -  Vs*1) 


and 


(44) 


3  loge  X 
“3^ 


;  knt-[s° 


■j2k 


-  (1  -  sk/nk) 


k=l 


/ 


-S'  k\ 

’  -  V  ") 


=  0. 


Solution  in  terms  of  and  q'2  is  accomplished  by  repeated  use  of 
equations  (34)  anti  (35)  . 

The  large  sample  2  by  2  variance-covariance  matrix  for  and 

a 2  is 
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where : 


and 


t 


fa  iog^  n 


da' 


2 


fa  loge  f 
&q,^3q'2 


/o  '°S„  x\ 
li - 2  J 

\  / 


(45) 


d  ‘  lcgt>  X 


dev 


2  lE'V 

k  =  l 


Sky 


nk(l  -  2,^  °2t) 


l  ( - 


^ 2  .  „  N  ks.  c 

5  loge  £  »— «  k 

So'jda^ 


-Q?,k 


E 

kal 


(4  6) 


a'  loge  £ 

-  2 

Sot, 


N  ,2  "a2k 

, — %  k  s,  c 


'1 


( 

k**JL  (; 


)  -  (ye 


-a2k\2 


Expanding  in  a  Taylor  series  in  and  up  Lo  and  inc 
terms  of  tlie  first  order,  a  special  case  of  (15),  wc  appre 
variance  of  ior  each  k  by  the  following  expression,  win 
in  [yj: 


(47)  Var  1<^  “  e 


-2«2k 


2,  2 


j^t'av  c^j  +  arjk  '  Var  0*2  euv  (a^ , 


lud  i ng 
ximate  the 
ch  appears 

V] 
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While  large-sample  theory  yields  asymptotic  normality  of  R^ , 

simulation  results  indicate  that  for  the  number  of  trial9  which  might 

reasonably  be  expected  in  a  development  program,  the  beta  distribution 

provides  a  better  (in  many  cases,  nearly  perfect)  approximation.  If 

it  is  assumed  that  the  parameters  of  the  beta  distribution  are  p,  and 

k 

q,  ,  by  the  method  of  moments,  we  tind  that 


(48) 


pk 


Rk<1 


V 


-  1 


Var  R, 


and 


(49) 


%  =  a  -  V 


V1  -  V 

Var  R, 


-  1 


Thus  a  lOOT-pcrccnt  lower  confidence  limit  for  R  is  given  by  L,  , 

K  ,  T 

where  it  the  solution  to  the  equation 


(50) 


/ 


k’T  r(pk  +  V  v1,,  ,  „  ,  , 

m x  (L'X)  d*  =  1-T 


v* 


This  development  is  found  in  [9]. 

Tf  the  maximum  likelihood  estimate  oi  a ,  is  negative,  8  is  set 

j  -  .  . 

equal  to  zero  and 


(51) 


Ilk  -  (1  -  a^, 


1 

i 


1 


i 


1 

1 
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in  which  there  is  no  reliability  growth.  Tlie  corresponding  estimation 
problem  is  then  not  very  interesting  or  enlightening  and  can  be  done 
without  difficulty. 

If  the  maximum  likelihood  estimate  of  ce.  causes  R.  to  become 

1  k 

negative,  we  then  put  =  1 .  Thus, 

-Q0k 

(52)  Rk  =  1  -  e  ‘  , 

and  the  function  we  maximize  is 


(53) 


-a2k\s 


-Q',k\nk-sk 


(54) 


iog^  X  =  const. 


k-l  '  '  k=l  N  ' 


Thus  , 


(55) 


d  log  X 


-a2k 


0ct2 


I  X  T-— "\  s  e  ■» — > 

f““D/  -  Z k(n^  v  • 


/  -or2k\ 
k-l  (l  -  e  J  k=l 


The  maximum  likelihood  estimator  Z  is  then  that  value  of  cr,  such  that 


■&2k  N 


2  -  2  k(nk  -  -v =  °* 


k-l 


(  '  ’  t-t 

n  -  c  J  k-i 


(56) 
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or 


(57) 


Lj  l  iFT  ‘  2  k("t  -  "W  ' 

k=l  -  1)  k-1 


Again  using  Newton' »  meLhod  to  iterate  on  a  solution  tor  a,? ,  we 
have  the  following:  if  S,  n_  j  is  the  (n  -  l)st  iteration  for  ci^,  th 


<53)  ^2 ,n  =  ®2 ,n- 1  + 


N 


s, 


k-1  (“2,0-!  _  )  k=l 


N  1 

-  E  k(nR  -  sk) 


N  ks  e 

E  k 

k= 


a,,  „  ,k 
1 ,  n-  1 


.  ,)2 


HODEf  3 

This  is  one  of  the  two  models  introduced  here.  We  assume  in 
this  case  that 


(59) 


f  (R  ,  or,  k)  =  R  -  a  , 


for  the  region  0  <  cf  <  R  1  >  tor  k  *-  1 ,  .  .  .  ,  N;  thus  for  the 

00 

region  0  <  cr  <  R  <1. 

°  CO 

From  (8)  and  (9)  we  have 


(60) 


1  no  r 

■'°C  ' 


N 


N 


dk 


u-  --  2; 


/n  -  c  't 

'  k  k' 


-  "  0, 


k-1  (K«  "  a  > 


k-1 


(1  -  K  +  t  ) 


dw 


N  .  k-1 

kskc 


EISk°  V" 

/!»  .V'\  /  1 


N  v  v  k-1 

kin,  -  s,  )  or 

■'  K 


k-1  (K»  "  a)  k-1 


( ]  -  H  .  +  a  ) 


cn 


(61) 


+ 


0. 
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Xho  problem  that  arises  here  and  that  is  brought  out  more  clearly  when 
we  apply  this  model  to  actual  reliability  predictions  is  that  we  have 
no  guarantee  (60)  and  (61)  will  have  unique  solutions  rhereby  enhancing 
the  difficulty  in  obtaining  the  maximum  likelihood  estimates.  This  is 
because  logc  £  is  not  necessarily  a  concave  function  of  R  and  a. 


io  demonstrate  this,  the  second  partial  derivative 


with  respect 


to  log^  £  are : 


(62) 


-  2  , 

6  l0£e  £ 


oft 


/rj  K2 

k-i  (R«  "  «  > 


uv 


V 


k=l  (1  -  \ 


k  ? 

'b  O'  ) 


(63) 


3"  log 


Y  * 


&R  'da 


Vv~l  N  ,  k-1 

( 1  .  V  j  k  2 

k=i  ( 1  K  +  G  ) 


and 


a  log  x 

(64)  - - f— 

dtr’ 


£ 

k~l 


b(k  -  !)„' 


k-; 


<n, 


•t.) 


■> 


d  -  K,  4  «*)  <«w  ..  *) 


b“  ] 


" 

2  N 

_ 

,C~'  . 

'  (’!k  “  Bk? 

k-1 

knk’  1 

,(R»  ' 

d  '  *-  +  **>  j 

and  it  follows  that  sut I icrcnt  conditions  guaranteeing  a  unique  maximum, 

, .  ,  .  .  .  d  log,.  £  h  leg  r 

=  0,  and 


uy  sul  v  mg  i  ne  equations'” - 

3H 

when 


rjQ 


“  0,  arc  violated 


d"  log  £ 


0 


-<Vk 


r  V  Z. 

S  lo8e  I  (nk  -  «k)a'1e 

~D  “  ^  ~Tf  -  aJk\2 

3a2  1-1  k2(l  -  2  ) 


Analogous  to  the  discussion  o£  Model  3,  we  can  state  the  following 
result.  If  cr^  and  are  s0  restricted  that. 


I  i  -I 

I  j 

|  ~l 


sulficient  conditions  guaranteeing  a  unique  maximum  by  solving 


CONFIDENCE  INTERVAI.3  FOR  MODELS  3  AND  4 


We  r.an  rewrite  Model  3  in  the  alternative  form 


(71) 


R  - 

00 


c-f* 


where  p  -  -log^  u  Thus,  by  the  invariance  principle  of  maximum 
likelihood  estimators 


=  R 


•Pk 


where  p  =  -loge  a-  By  (15), 

(72)  Var  R.  =  Vur  R  +  k2c  2^k  Var  p  -  2ke  cov  (R  ,  p) . 

x  k  CO  r  UJ 


Thus,  an  approximate  lOOT-pcrcent  lower  confidence  limit  for  R^  (the 
predicted  reliability  at  the  k-th  stage  of  testing)  is  given  by 


(73)  Xk 


R.  -  Z.  Jv^r  R, 
k  1-t  i!  k 

R.  -  Z,  J~Var  R  4  kV2f3k  Var  p  -  2ke~pk  cov  (R  ,  p)  , 
*'  1  T  *  -  ^ 


where  Var  R  ,  cov  (R  ,  R)  and  Var  g  are  obtained  in  the  usual  manner 

CO  CO 

and  Z,  is  the  solul.  ion  of  the  equation 
1-  T 


T- 


The  development  of  the  lower  confidence  limit  for  Model  4  is 
completely  analogous  with  a  different  value  of  Var  R^.  For  Model  4 
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(74)  Var  R 


^o-^/k  2Q'j 

e  Var  o^  -I - ^  Vav  a.^  ~  cov  (a^.  o^) 

k 


ALTERNATIVE  BAYESIAN  PROCEDURE  FOR 
OBTAINING  LOWER  CONFIDENCE  INTERVALS 

The  test;  data  arc  divided  into  N  stages  by  some  predetermined 
criterion  such  as  design  change.  Since  flight  test  data  are  of  the 
go,  no-go  variety,  the  probability  that  exactly  x^  successes  are 
recorded  in  n.  trials  at  Stage  1  is 


/S\  xi  nrxi 

(75)  -(hJ’I  <»  -  V 


Xj  1=5  0,  1,  •  •  •  ,  n  ^  . 


Wc  shall  assume  that  R  lias  a  beta  density  with  parameters  a^  and  b^ 
as  its  prior  probability  density  function  (pdf).  TliaL  is, 


(76)  f12(Rl) 


|l(a  +  b,)  a  -1  b  -1 

- - - —  p  1  (i  .  n  i  1 

Irca^rtbj)  Ki  u  V 


0  £  <;  1 


elsewhere. 


If  we  have  no  previous  information  regarding  the  system's  relia¬ 
bility  at  Stage  1,  we  must  choose  the  parameters  aj  and  b^  subjectively 
We  shall  use  Lhe  method  Fox  [4]  describes.  First  of  all,  we  let  R  be 
our  subjective  estimate  of  .  If  we  agree  Lhai  our  subjective  esti¬ 
mate  should  be  our  most  likely  estimate  wc  then  set 


al  '  1 
R,  -  ~  :: 

ax  +  b  -  2 
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assuming  that  (77)  is  not  the  uniform  prior  density  on  (0,  1).  (If 
(77)  is  taken  as  the  uniform  prior  density  the  modification  to  be 
made  is  discussed  at  the  end  ol  the  subsection.)  We  then  ask  what 
the  odds  are  that  the  true  value  of  R^  will  lie  in  the  interval 
(R^  -  kk^,  R^  d-  kR^),  where  k  is  predetermined?  If  we  set  these 
odds  at  x  to  y,  we  can  express  this  mathematically  as 


(78) 


„R,+kR 


/“'l  ~'l 

J  _  I12<«,) 


R1-kR1 


v, 


where  v  -  x  y  •  Tills  is  Fox's  equation  [4,  Eq .  (1.6),  p.  3].  From 
(77)  and  (78),  witli  the  aid  of  the  tables  in  [4],  we  can  now  determine 
a^  and  b^. 

To  obtain  a  Bayes Lau  lower  confidence  interval  for  Rj  after  we 
have  completed  Stage  r,  wo  note  that  the  posterior  pdl  of  R^  ,  given 
x, ,  is  defined  by 

a. 


(79)  f13(k1lx1)  = 


T (a,  d  b,  +  n, ) 

i  i  j 


r(a1  d  x1)r(b1  d  (iij  -  x  / 


a,+x,-l  b.d-(n.-x.) 

k,j  J  a  -  up  ‘  '  1  L 


1 


Thus,  a  (1  -  cr^)  -  100-pcrcont  level  Bayesian  lower  confidence  interval 
for  Rj  is  given  by 


(80) 


fu(V 


xi) 


JR, 


1  ’ 


wiiere  is  so  chosen  that  (80)  holds. 
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Following  the  method  of  derivation  for  Stage  1,  we  can  now 
obtain  a  (1  -  q'  )  •  100-percent  level  Bayesian  lower  confidence 
interval  for  R2  in  an  analogous  manner.  We  use  the  data  in  Stage  1, 
however,  to  require  that  the  most  likely  estimate  for  R2  is  given  by 


(81) 


R2  " 


a2  -  1 


a]  +  *1  -  1 


a2  +  b2  "  2  aJ  +  bl  +  nl  '  2 


Thus,  by  this  meLhod  we  obtain  successive  Bayesian  estimates  of 
reliability  growth,  at  each  stage  and  their  associated  lower  confi¬ 
dence  bounds.  There  appears  to  be  no  way  in  which  we  can  use  the 
Bayesian  approach  to  obtain  a  lower  confidence  bound  at  the  (N  4-  l)rh 
stage  (i.e.,  for  the  predicted  reliability  at  Stage  N  4  1  of  our 
testing  program) . 

The  procedure  described  above  needs  slight  modification  if  we 
assume  e  uniform  density  for  .  (In  this  case,  we  assume  total 
ignorance  of  the  system's  reliability  prior  to  Stage  1.  That  is, 
any  one  value  of  P.^  is  as  likely  to  occur  as  any  other  value  prior 
to  starting  the  test  program.)  In  Litis  instance  we  know  that 
al  ~  *“  1>  which  in  turn  yields 


r(n^  4-  2) 


(82)  rUi  +  1)r(tlj  .  Xi  q.  D  % 


■i  nrxi 

H,  (1  -  R.) 


We  then  proceed  as  before  to  obtain  confidence  intervals  at  Stage  1 
and  the  succeeding  stages  for  which  data  arc  available. 
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